The processing of WM information during delay periods has been studied extensively using neural recordings (for reviews, see refs. 35,49,50,51,52,53). Here, using novel stimulus materials and tailored geometry analyses, we showed that miniature gaze deflections can disclose an array of WM-associated phenomena that to the best of our knowledge was previously only observed in neural signals, including (1) a sustained encoding of the task-relevant stimulus feature, which (2) shows a different format than during perception, can (3) persist while also encoding new perceptual information, (4) ramps up throughout delay periods when relevant for an upcoming test and (5) returns to baseline when uncued (or ‘unattended’). Beyond this, the gaze geometries indicated that temporary inattention rendered the WM information more generalized (object-independent) and potentially more categorically biased. These format changes during maintenance were similarly observed when attention to the memorandum was withdrawn by explicit retro-cueing or by presenting further WM information.
Behaviourally, our results replicate and extend previous findings that temporary inattention renders working memories less precise and more biased37,38. The eye-tracking results shed light on the temporal unfolding of potentially underlying format changes during WM storage. The gaze patterns during perceptual processing were clearly object-specific, indicating a focus on concrete visual details. When the last-seen stimulus (Stimulus 2) was immediately cued (with an auditory retro-cue that offered no visual distraction), some of this object specificity was sustained throughout the ensuing WM delay. In contrast, for the first-presented stimulus (Stimulus 1), the object specificity dropped abruptly as soon as Stimulus 2 processing commenced. Of note, Stimulus 1 encoding did continue throughout Stimulus 2 processing. However, its format changed to object-independent (or ‘abstract’) during the object-specific (or ‘concrete’) encoding of Stimulus 2—as if the memory of Stimulus 1 was reformatted to ‘evade’ the format of the currently perceived stimulus. Subsequent cueing did not revert this effect, nor did we find any re-emergence of object specificity after unattended storage, for either stimulus, in the second delay. Together, these results support the idea that temporary (or partial) inattention may render the task-relevant WM information (here, orientation) increasingly less ‘concrete’ (visual-sensory) and more generalized or ‘abstract’.
Further support for this idea comes from our analysis of the cardinal repulsion bias. In parallel with the object independence of gaze patterns, the repulsive cardinal reporting bias increased with the time a given stimulus had been temporarily (or partially) unattended. Repulsive-orientation bias in WM tasks has been explained, for example, by efficient coding principles, in terms of relatively finer tuning to cardinal orientations, reflecting their relative prevalence in natural environments18. An alternative framing of the cardinal repulsion biases in our experiment with real-life objects could be in terms of more explicit semantic categorization (for example, ‘left’/‘right’ and ‘up’/‘upside-down’)17,54. The results may thus also reflect increased reliance on semantics39 and/or (pre)verbal labels when restoring information from unattended storage34, which would be in line with a higher level of abstraction. Although our geometrical analysis approach is agnostic to the mechanistic cause of cardinal biases, we found some indications that they were also evident in biased gaze geometries during the stimulus-free retention periods (for related findings in neuroimaging, see refs. 22,23,55). The latter result was statistically weak and should be revisited in future work, possibly under conditions that induce even stronger biases in behaviour (for example, higher WM loads)18.
A remarkable aspect of our results is the small amplitude of the eye movements that disclosed such rich information. The mass of the raw position samples in our analysis were within a <1° visual angle around fixation (Fig. 1c). A discernible ‘circular’ structure in averaged data points (Fig. 1d) measured only ~0.2–0.3° in diameter, which is near the eye-tracker’s accuracy limit, and was only a fraction of the memory items’ physical size. Together with our online fixation control (Methods), these descriptives render it unlikely that our results were attributable to reflexive saccades to the location of peripheral stimulus features. Further analysis (Fig. 5) indicated that the findings more likely reflect microsaccadic activity during attempted fixation. Systematic microsaccade patterns have previously been linked to covert spatial attention29,48, indicating that in the present context, they might have reflected mental orienting towards a spatial coordinate or direction30,31. Together, our results indicate that participants generally oriented attention towards the objects’ real-life ‘top’, but with varying degrees of bias towards specific object features (resulting in object-specific orientation patterns) and/or away from cardinal axes (resulting in cardinal repulsion).
Under a view of the miniature gaze patterns reflecting covert spatial attention, our analysis tracked with high temporal resolution the time course of attention allocation to WM information in a dual retro-cue task. Before cueing, encoding a new stimulus (Stimulus 2) did not immediately eradicate or replace the attentional orienting to the previous stimulus (but did change its qualitative format; see above). At face value, the temporary simultaneity of both WM contents (Fig. 2e), in a putative index of attention, might seem to be at odds with the idea of an exclusive single-item focus of attention in WM56,57 (but see refs. 58,59). However, another possible interpretation is that the (re-)allocation of attention to different stimuli (or tasks) in WM may take time to complete. For instance, the encoding of the uncued stimulus fully returned to baseline only ~0.5–1.5 s after the cue, which is broadly consistent with previous behavioural and EEG work on the time course of WM-cueing effects27,60,61,62. Compared to this, the reformatting into a more generic, object-independent format was rapid, both for Stimulus 1 when encoding Stimulus 2 (see above) and for Stimulus 2 itself when it was uncued. Consistent with these results, a recent study found that low-level perceptual bias induced by concurrent WM information (cf. ref. 63) dissipated quickly with new visual input44. These findings are in line with adaptive format changes in WM, potentially providing fast protection from interference beyond the overall reallocation of attention between different stimuli and/or tasks.
Previous WM studies using retro-cues yielded mixed results about potential costs for the first of two successively presented stimuli. Using visual retro-cues, one study found no differences between visual gratings presented first or second in either behaviour or functional imaging-decoding during the WM delay8, which has been taken as evidence that intervening stimuli may cause little to no interference for visual WM representations9. Another study, using visual retro-cues with tactile WM stimuli, did find lower performance for the first stimulus27, a finding we replicated here with auditory cueing of visual WM information. One possibility is that different-modality cues (for example, auditory when the WM stimuli were visual) interfere less with the short-term memory of the last-presented stimulus than same-modality cues would (for example, visual cues with visual WM stimuli)38. Different-modality cues may thus leave the memory trace of the last-presented stimulus more intact compared to the first stimulus (which is always followed by the same-modality input of the second stimulus). This aside, the format changes induced by the intervening stimulus were qualitatively similar to those after unattended storage, in line with a common explanation in terms of temporarily withdrawn attention.
Our findings of increasingly more object-independent gaze geometries do not rule out that the brain may maintain detailed visual memories in ways that would not register in eye tracking. More generally, we can only speculate whether the minuscule eye movements observed in our experiment played a functional role or whether they were merely epiphenomena of other processes. We consider it possible that our paradigm promoted aspects of WM-related processing to become visible at the surface of ocular activity, but that the ocular activity itself may have had little or no direct role in the WM processing proper (for related discussion, see refs. 30,47,64; but see refs. 65,66 for a role of eye movements in episodic memory retrieval). This speculation also takes note of several recent failures to decode visuospatial WM information from eye tracking, most notably in control analyses supplemental to neural decoding, where systematic eye movements were ruled out as a potential confound12,67,68,69 (but see refs. 32,33,70). At the same time, our findings sound a cautionary note that stimulus-dependent eye movements in visual WM tasks can be very small, hard to prevent, persistent and, above all, informative.
In summary, despite discouraging participants from eye motion through closed-loop fixation control, we found the orientation of visual objects robustly reflected in miniature gaze patterns during cued WM maintenance. The geometry of the gaze patterns underwent systematic changes, indicating that temporary inattention increased the level of abstraction (and categorical bias) of the information in WM. Stimulus-dependent eye movements may not only pose a potential confound but also be a valuable source of information in studying visuospatial WM.
